Complexity of training ReLU neural network
نویسندگان
چکیده
In this paper, we explore some basic questions on the complexity of training neural networks with ReLU activation function. We show that it is NP-hard to train a two-hidden layer feedforward network. If dimension input data and network topology fixed, then there exists polynomial time algorithm for same problem. also if sufficient over-parameterization provided in first hidden network, which finds weights such output over-parameterized matches given data.
منابع مشابه
Improving performance of recurrent neural network with relu nonlinearity
In recent years significant progress has been made in successfully training recurrent neural networks (RNNs) on sequence learning problems involving long range temporal dependencies. The progress has been made on three fronts: (a) Algorithmic improvements involving sophisticated optimization techniques, (b) network design involving complex hidden layer nodes and specialized recurrent layer conn...
متن کاملscour modeling piles of kambuzia industrial city bridge using hec-ras and artificial neural network
today, scouring is one of the important topics in the river and coastal engineering so that the most destruction in the bridges is occurred due to this phenomenon. whereas the bridges are assumed as the most important connecting structures in the communications roads in the country and their importance is doubled while floodwater, thus exact design and maintenance thereof is very crucial. f...
Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network
We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normalization (BN) technique actually adapts the “normalized” bias such that it app...
متن کاملTraining Better CNNs Requires to Rethink ReLU
With the rapid development of Deep Convolutional Neural Networks (DCNNs), numerous works focus on designing better network architectures (i.e., AlexNet, VGG, Inception, ResNet and DenseNet etc.). Nevertheless, all these networks have the same characteristic: each convolutional layer is followed by an activation layer, a Rectified Linear Unit (ReLU) layer is the most used among them. In this wor...
متن کاملA New Approach for Investigating the Complexity of Short Term EEG Signal Based on Neural Network
Background and purpose: The nonlinear quality of electroencephalography (EEG), like other irregular signals, can be quantified. Some of these values, such as Lyapunovchr('39')s representative, study the signal path divergence and some quantifiers need to reconstruct the signal path but some do not. However, all of these quantifiers require a long signal to quantify the signal complexity. Mate...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Discrete Optimization
سال: 2022
ISSN: ['1873-636X', '1572-5286']
DOI: https://doi.org/10.1016/j.disopt.2020.100620